Measuring stability of feature ranking techniques: a noise-based approach

One very common criterion used to evaluate feature selection methods is the performance of a chosen classifier trained with the selected features. Another important evaluation criterion that has, until recently, been neglected is the stability of these feature selection methods. While other studies have shown interest in measuring the degree of agreement between the outputs of a technique trained on randomly selected subsets from the same input data, this study presents the importance of evaluating stability in the presence of noise. Experiments are conducted with 17 filters (six standard filter-based ranking techniques and 11 threshold-based feature selection techniques) on nine different real-world datasets. This paper identifies the techniques that are inherently more sensitive to class noise and demonstrates how certain characteristics (sample size and class imbalance) of the data can affect the stability performance of some feature selection methods.